Click any tag below to further narrow down your results
Links
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.
+ northguard
+ xinfra
+ kafka
+ linkedin
+ distributed systems
metadata management
+ storage architecture
Marquez is an open-source metadata server that collects and visualizes data lineage across your organization's pipelines. It works with major tools like Airflow, Spark, dbt, and Dagster to show you where data comes from, where it goes, and how jobs depend on each other.
- Real-time metadata collection through an OpenLineage-compatible API endpoint that integrates with existing data orchestration tools
- Web UI displays data dependencies and lineage as a visual graph, letting you trace datasets back through pipelines and see job inputs/outputs
- Lineage API enables automation for tasks like backfills and root cause analysis by letting you query dependencies across multiple platforms